Telerobotic System with a Dual Application Screen Presentation

ABSTRACT

A proctoring system that includes a communication device coupled to a remote station. The remote station has a visual display that displays first information relating to an action that causes an effect on an object, and simultaneously displays second information relating to the effect on the object. The remote station includes at least one input device that allows a communication to be transmitted by an operator to the communication device. By way of example, during the deployment of a heart stent, a specialist doctor may remotely view real-time fluoroscopy imagery and patient hemodynamics. The specialist can remotely proctor medical personnel on the proper orientation and timing requirements for installing the stent.

BACKGROUND

1. Technical Field

The subject matter disclosed generally relates to the field ofproctoring.

2. Related Art

Robots have been used in a variety of applications ranging from remotecontrol of hazardous material to assisting in the performance ofsurgery. For example, U.S. Pat. No. 5,762,458 issued to Wang et al.discloses a system that allows a surgeon to perform minimally invasivemedical procedures through the use of robotically controlledinstruments. One of the robotic arms in the Wang system moves anendoscope that has a camera. The camera allows a surgeon to view asurgical area of a patient.

There has been marketed a mobile robot by InTouch Technologies, Inc.,the assignee of this application. The InTouch robot is controlled by auser at a remote station. The remote station may be a personal computerwith a joystick that allows the user to remotely control the movement ofthe robot. Both the robot and remote station have cameras, monitors,speakers and microphones that allow two-way video/audio communicationbetween the remote station operator and personnel of the robot site.

The InTouch system has been used to remotely assist and train medicalpersonnel. For example, a medical specialist at one location can viewthe patient at a remote location and provide proctoring to medicalpersonnel at the patient site. It would be desirable to enhance theproctoring capability of the InTouch system.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an illustration of a robotic system;

FIG. 2 is a schematic of an electrical system of a robot; and,

FIG. 3 is a display user interface of a remote station having a firstscreen field and a second screen field.

DETAILED DESCRIPTION

Disclosed is a proctoring system that includes a communication devicecoupled to a remote station. The remote station has a visual displaythat displays first information relating to an action that causes aneffect on an object, and simultaneously displays second informationrelating to the effect on the object. The remote station includes atleast one input device that allows a communication to be transmitted byan operator to the communication device. By way of example, during thedeployment of a heart stent, a specialist doctor may remotely viewreal-time fluoroscopy imagery and patient hemodynamics. The specialistcan remotely proctor medical personnel on the proper orientation andtiming requirements for installing the stent. The fluoroscopy imagerydisplays an image of the stent being inserted into a beating heart. Thestent will cause an effect on the heart such as a change in heart beat.The hemodynamics provide data regarding the beating heart, which is theeffect of the stent. The simultaneous display of patient hemodynamicsand fluoroscopy imagery provide the specialist with real-time data sothat they can issue a command such as “deploy now” to insert the stent.Alternatively, the proctoring specialist would converse with theintra-operative surgeon to confer on the pattern of hemodynamicinformation being received and suggest through this information that “aswe can observe this is the ideal timing of implant deployment”.

Referring to the drawings more particularly by reference numbers, FIG. 1shows a system 10. The robotic system includes a robot 12, a basestation 14 and a remote control station 16. The remote control station16 may be coupled to the base station 14 through a network 18. By way ofexample, the network 18 may be either a packet switched network such asthe Internet, or a circuit switched network such has a Public SwitchedTelephone Network (PSTN) or other broadband system. The base station 14may be coupled to the network 18 by a modem 20 or other broadbandnetwork interface device. By way of example, the base station 14 may bea wireless router. Alternatively, the robot 12 may have a directconnection to the network thru for example a satellite.

The remote control station 16 may include a computer 22 that has amonitor 24, a camera 26, a microphone 28 and a speaker 30. The computer22 may also contain an input device 32 such as a joystick or a mouse.The control station 16 is typically located in a place that is remotefrom the robot 12. Although only one remote control station 16 is shown,the system 10 may include a plurality of remote stations. In general anynumber of robots 12 may be controlled by any number of remote stations16 or other robots 12. For example, one remote station 16 may be coupledto a plurality of robots 12, or one robot 12 may be coupled to aplurality of remote stations 16, or a plurality of robots 12.

Each robot 12 includes a movement platform 34 that is attached to arobot housing 36. Also attached to the robot housing 36 are a camera 38,a monitor 40, a microphone(s) 42 and a speaker(s) 44. The microphone 42and speaker 30 may create a stereophonic sound. The robot 12 may alsohave an antenna 46 that is wirelessly coupled to an antenna 48 of thebase station 14. The system 10 allows a user at the remote controlstation 16 to move the robot 12 through operation of the input device32. The robot camera 38 is coupled to the remote monitor 24 so that auser at the remote station 16 can view a patient. Likewise, the robotmonitor 40 is coupled to the remote camera 26 so that the patient mayview the user. The microphones 28 and 42, and speakers 30 and 44, allowfor audible communication between the patient and the user.

The remote station computer 22 may operate Microsoft OS software andWINDOWS XP or other operating systems such as LINUX. The remote computer22 may also operate a video driver, a camera driver, an audio driver anda joystick driver. The video images may be transmitted and received withcompression software such as MPEG CODEC.

The robot 12 may be coupled to one or more medical monitoring devices50. The medical monitoring device 50 can take medical data from apatient. By way of example, the medical monitoring device 50 may be astethoscope, a pulse oximeter and/or an EKG monitor. The medicalmonitoring device 50 may contain a wireless transmitter 52 thattransmits the patient data to the robot 12. The wirelessly transmitteddata may be received by antennae 46, or a separate antennae (not shown).The robot 12 can then transmit the data to the remote station 16. Themedical devices 50 may include a fluoroscope that can generatefluoroscopy images. Additionally, the medical devices may generatepatient hemodynamic data such as heart rate.

The wireless transmission from the medical monitoring device 50 may bein accord with various wireless standards such as IEEE. The standardused to transmit data from the medical monitoring device 50 should notinterfere with the wireless communication between the robot 12 and thebase station 14. Although wireless transmission is shown and described,it is to be understood that the medical monitoring device 50 can becoupled to the robot 12 by wires (not shown).

When transmitting fluoroscopy imagery and patient hemodynamics, thesystem can combine hemodynamic data with a frame of the fluoroscopyimagery into a double-wide frame that is transmitted to the remotestation for split screen display. The stitching of the two frames wouldoccur prior to compression, enabling the compression algorithm (e.g.MPEG-4) to utilize redundant information that may occur between the twoframes. This potentially will result in equivalent quality and lowerbandwidth as compared to two streams compressed and transmittedseparately. Alternatively, the fluoroscopy imagery and hemodynamics canbe transmitted in a decoupled manner and then combined with timinginformation such as time stamps that are inserted into the informationbefore transmission.

The remote station 16 may be coupled to a server 54 through the network18. The server 54 may contain electronic medical records of a patient.By way of example, the electronic medical records may include writtenrecords of treatment, patient history, medication information, a medicalimage, such as an e-ray, MRI or CT scan, EKGs, laboratory results,physician notes, etc. The medical records can be retrieved from theserver 54 and displayed by the monitor 24 of the remote station 16. Inlieu of, or in addition to, the medical records can be stored in themobile robot 12. The remote station 16 may allow the physician to modifythe records and then store the modified records back in the server 54and/or robot 12.

FIG. 2 shows an embodiment of a robot 12. Each robot 12 may include ahigh level control system 60 and a low level control system 62. The highlevel control system 60 may include a processor 64 that is connected toa bus 66. The bus is coupled to the camera 38 by an input/output (I/O)port 68, and to the monitor 40 by a serial output port 70 and a VGAdriver 72. The monitor 40 may include a touchscreen function that allowsthe patient to enter input by touching the monitor screen.

The speaker 44 is coupled to the bus 66 by a digital to analog converter74. The microphone 42 is coupled to the bus 66 by an analog to digitalconverter 76. The high level controller 60 may also contain randomaccess memory (RAM) device 78, a non-volatile RAM device 80 and a massstorage device 82 that are all coupled to the bus 72. The mass storagedevice 82 may contain medical files of the patient that can be accessedby the user at the remote control station 16. For example, the massstorage device 82 may contain a picture of the patient. The user,particularly a health care provider, can recall the old picture and makea side by side comparison on the monitor 24 with a present video imageof the patient provided by the camera 38. The robot antennae 46 may becoupled to a wireless transceiver 84. By way of example, the transceiver84 may transmit and receive information in accordance with IEEE 802.11b.The transceiver 84 may also process signals from the medical monitoringdevice in accordance with IEEE also known as Bluetooth. The robot mayhave a separate antennae to receive the wireless signals from themedical monitoring device.

The controller 64 may operate with a LINUX OS operating system. Thecontroller 64 may also operate MS WINDOWS along with video, camera andaudio drivers for communication with the remote control station 16.Video information may be transceived using MPEG CODEC compressiontechniques. The software may allow the user to send e-mail to thepatient and vice versa, or allow the patient to access the Internet. Ingeneral the high level controller 60 operates to control communicationbetween the robot 12 and the remote control station 16. The low levelcontroller can control movement of the robot 12. The system may be thesame or similar to a robotic system provided by the assignee InTouchTechnologies, Inc. of Santa Barbara, Calif. under the name RP-7, whichis hereby incorporated by reference.

FIG. 3 shows a visual display 120 of the remote station. The visualdisplay 120 displays a first screen field 122 and a second screen field124. The two screen fields may be created by two different monitors.Alternatively, the two screen fields may be displayed by one monitor.The first and second screen fields 122 and 124 may be part of anapplication program(s) stored and operated by the computer 22 of theremote station 16.

The first screen field 122 can display first information that relates toan action that causes an effect on an object. The second screen field124 can simultaneously display second information relating to the effecton the object. By way of example, the system can be used for proctoringduring a medical procedure. The first screen field 122 may displayfluoroscopy imagery showing a stent being inserted through the aorta ofa beating heart. The second screen field 124 may display patienthemodynamics such as heart rate. It is critical for the stent to beinserted in a proper position and during certain timing sequences withthe beating of the heart. A medical specialist at the remote station cansimultaneously view both the position of the stent and the hemodynamicdata. This allows the specialist to proctor someone located at thepatient location as to the timing of certain stent movement. Proctoringcan be accomplished by video and/or audio communication transmitted fromthe remote station to the robot. Although insertion of a stent is shownand described, the system can be used in any type of procedure such asthe deployment of a clot-buster as an example.

The visual display 120 may include a latency indicator 126 that providesthe latency between the sending and the receipt of the informationdisplayed in the two synchronized screen fields 122 and 124. Forexample, FIG. 3 shows a latency of 92 milliseconds between the sendingof fluoroscopy imagery and hemodynamic data from the OR, and receipt ofthe imagery and data at the remote station. Alternatively, the latencyvalue may indicate the round trip latency from OR to remote station andback to the OR, hence representing the time between capture of an imagein the OR and receipt of a related proctor instruction in the OR. Thislatency information may be critical to deployment, as the remote expertmust be certain that when stating a command such as ‘deploy now’, themessage is arriving at the proper moment, rather than at aninappropriate time later. The visual display 120 may also include anindicator 128 that provides an indication of when the latency exceeds athreshold. FIG. 4 shows an indication of “Safe Timing” which isdisplayed when the latency is below the threshold. In one embodiment, aphysician may select the type of procedure being proctored from adrop-down menu (not shown), and the threshold value, which representsthe maximum safe latency to provide timing-sensitive instructions, willbe set from a table of values indexed by procedure type.

The system allows an operator to simultaneously view information thatrelates to an action that will affect an object and information thatrelates to the effect on the object. This allows the operator to providecommunication to the site of the object, for example a doctor mayproctor another doctor at a surgical site.

There are multiple types of procedures that may benefit from thistechnology. In addition to the transcatheter heart valve deploymentdescribed above, the system may be used with a Deep Brain Stimulation(DBS) procedure. The patient's skull is prepared for neurosurgery and isawake, while the surgeon drills through the skull and placesscan/stimulation probes in a specific targeted area of the brain. Thesurgeon simultaneously reads the DBS monitor signs (waveforms andnumbers) and then asks the patient to hold up an arm and count down from100. The patient may struggle with this task. The surgeon may amplifythe stimulation to the targeted area of the brain and ask the patientrepeat this task until the patient can perform the task with greaterease. In this procedure, the proctoring surgeon must simultaneously see:

(a) the head of the bed where the patient's head is presented, includingthe local surgeon performing drilling and stimulation;

(b) the DBS monitor relative to what the local surgeon is doing; andpossibly

(c) a plan view of the operative field.

In usage, a physician may be in transit when receiving a call requestingproctoring, and may begin the proctoring session while walking to theoffice, utilizing a mobile device such as an iPad. On the mobile displaythere may only be space for the flouroscopy imagery, which may besufficient for proctoring during the initial segment of the procedureonly. Upon arrival at the office, the physician may disconnect from themobile device, which may use 3G and/or 802.11 connectivity, andre-connect from a wide-screen desktop system on an Ethernet connection.The physician may now have high-resolution views of both fluoroscopy andhemodynamics, and may proctor the final deployment. This transitionallows for continuity of care across the two devices. Alternatively, thephysician may begin using the desktop system while still connectedthrough the mobile device, and the system would effect a seamlessautomatic disconnect from the mobile device and re-connect to thedesktop system.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of and not restrictive on the broad invention, andthat this invention not be limited to the specific constructions andarrangements shown and described, since various other modifications mayoccur to those ordinarily skilled in the art.

1. A proctoring system, comprising: a communication device; and, aremote station that is coupled to said communication device, said remotestation having a visual display that displays first information relatingto an action that causes an effect on an object and simultaneouslydisplays second information relating to the effect on the object, saidremote station including at least one input device that allows acommunication to be transmitted by an operator to said communicationdevice.
 2. The system of claim 1, wherein said communication deviceincludes a monitor and a speaker.
 3. The system of claim 2, wherein saidcommunication device includes a mobile platform that is controlledthrough said remote station.
 4. The system of claim 1, wherein saidfirst information includes a fluoroscopy image and said secondinformation includes at least one hemodynamic data.
 5. The system ofclaim 1, wherein said remote station displays a latency that representsa network delay between a local capture of said first information andsaid second information, and a remote display of said first and secondinformation.
 6. The system of claim 1, wherein said remote stationdisplays an indicator that relates to whether a latency exceeds athreshold.
 7. The system of claim 1, wherein said at least one inputdevice of said remote station includes a camera and a microphone.
 8. Thesystem of claim 1, further comprising a network that couples saidcommunication device to said remote station.
 9. A proctoring system,comprising: a robot that includes a camera, a monitor, a speaker and amicrophone; and, a remote station that is coupled to said robot, saidremote station having a visual display that displays first informationrelating to an action that causes an effect on an object andsimultaneously displays second information relating to the effect on theobject, said remote station including a camera that is coupled to saidrobot monitor, a monitor that is coupled to said robot camera, amicrophone that is coupled to said robot speaker and a speaker that iscoupled to said robot microphone.
 10. The system of claim 9, whereinsaid robot includes a mobile platform that is controlled through saidremote station.
 11. The system of claim 9, wherein said firstinformation includes a fluoroscopy image and said second informationincludes at least one hemodynamic data.
 12. The system of claim 9,wherein said remote station displays a latency that represents a networkdelay between a local capture of said first information and said secondinformation, and a remote display of said first and second information.13. The system of claim 9, wherein said remote station displays anindicator that relates to whether a latency exceeds a threshold.
 14. Thesystem of claim 9, further comprising a network that couples saidcommunication device to said remote station.
 15. A method forproctoring, comprising: displaying at a remote station first informationrelating to an action that causes an effect on an object andsimultaneously displaying at the remote station second informationrelating to the effect on the object; and, transmitting a communicationfrom an operator of the remote station to a person involved with theaction.
 16. The method of claim 15, wherein the transmittedcommunication includes an image of the operator and a voicecommunication from the operator.
 17. The method of claim 16, furthercomprising displaying the image on a robot monitor and emitting thevoice communication from a robot speaker.
 18. The method of claim 17,further comprising moving the robot through an input of the remotestation.
 19. The method of claim 15, wherein the first informationincludes a fluoroscopy image and the second information includes atleast one hemodynamic data.
 20. The method of claim 15, furthercomprising displaying a latency that represents a network delay betweena local capture of said first information and said second information,and a remote display of said first and second information.
 21. Themethod of claim 15, further comprising displaying an indicator thatrelates to whether a latency exceeds a threshold.
 22. A method formedical consultation, comprising: transmitting an image of a patientfrom a telehealth device; displaying the image of a patient at a firstremote station; initiating a connection between the telehealth deviceand a second remote station; and, displaying the image of the patientwith additional information at the second remote station.
 23. The methodof claim 22, wherein the telehealth device is a robot.
 24. The method ofclaim 22, wherein the image of the patient displayed by the first remotestation is a flouroscopy image.
 25. The method of claim 22, wherein theadditional information is hemodynamics data.
 26. The method of claim 22,wherein the additional information is a larger resolution video image.27. The method of claim 22, wherein communication with the first remotestation view is terminated upon initiation of the second remote stationconnection.
 28. A system, comprising: a communication device thatcombines a video frame with additional information, compresses saidcombination of said video frame and additional information and transmitsthe compressed combination of said video frame and said additionalinformation; and, a remote station that is coupled to said communicationdevice and receives said compressed combination of said video frame andsaid additional information.
 29. The system of claim 28, said videoframe includes a fluoroscopy image and said additional informationincludes at least one hemodynamic data.
 30. The system of claim 28,wherein said communication device includes a monitor and a speaker. 31.The system of claim 30, wherein said communication device includes amobile platform that is controlled through said remote station.